Repository navigation
fix(service): Cap known values of runs without a token, key write-token runs on the main thread - #48
Merged
Merged
Conversation
…en runs on the main thread Address the review of #45: - Count `known` across all sessions of a run without the token (10,000 values, 256 Ki characters). A request past it fails with 400 and ends the run. No value is dropped, so no key changes. - Key and redact runs with the token on the main thread, as on main. The worker serves runs without the token only, with a 256 MiB old space and a 4 MiB semi-space. - Leave values over 32,767 characters out of `known` and templates. V8 cannot run a pattern with them, so those requests failed before. - Count matches in `extractValues()` as it finds them, so the cap of 100,000 values stops before memory grows. - Write the key text of a body without a copy of it. - Group IPv4-mapped IPv6 addresses in all forms as IPv4. - README: split deploy order for the PROXY header, the /48 gap. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Address the review of #48: - Runs without the token share a budget of 128 MiB of request bodies. A request holds its bytes from when they come until it finishes, also while its key waits for the key worker. A request that does not fit gets 503 with `Retry-After: 1`, and the run goes on. A client that declares a large body and then stops holds only what it sent. 30 and 60 parallel 16 MiB bodies now peak at about 1 GB of RSS for the service and the worker, down from 1.9 GB and 3.4 GB. - README: the deploy order resets the VM twice, and a reset ends open runs. Document the body budget. - Fix the doc comment of `canBeKnown()`: V8 fails on any text. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
…t slow bodies One client that started a run without a token could hold all 128 MiB of the body budget: 8 tunnels that each sent 16 MiB - 1 KiB and then one byte now and then. Node does not time out requests in a tunnel, because the tunnel server does not listen, so nothing gave the bytes back. Every other run without a token then got HTTP 503. - The runs of one client (`clientGroup()` of the client that started the run) can hold 32 MiB of the budget. - A body of a run without a token that does not come in 30 s gets HTTP 408, the connection closes, and the bytes go back. - Count a 503 as `roach.request` with `roach.result:busy`, and a 408 as `roach.result:timeout`, and add a TELEMETRY.md query. - README: the new limits, the memory bound for write bodies of a normal size, and the gap that Node does not time out tunneled requests. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
dcramer
marked this pull request as ready for review
November 5, 2026 15:33
sentry-junior Bot
added a commit
that referenced
this pull request
Nov 5, 2026
Keep both sides of #48 and of this branch: - server.ts: `readBody` clears the 30 s read deadline on end and on an error, and an ECONNRESET of a request still becomes the 400 `aborted` error. `answerError` counts `aborted`, `busy` (503), and `timeout` (408) as `roach.request`. The comment of the read deadline now says why it is shorter than Node's 300 s. - service.test.ts: keep both sets of helpers. `startBody` uses `openTunnel`. The aborted test also checks that a closed request gives its bytes of the budget back. - README.md: one metric row with all results. Remove the gap about Node time limits, because this branch fixes it. - TELEMETRY.md: Busy, Timeout, and Aborted, and leave all three out of the hit rate.
dcramer
added a commit
that referenced
this pull request
Nov 5, 2026
CI already runs on push to `main` (`ci.yml` has `push: branches: [main, "release/**"]`). But GitHub started no push workflows for `09f6582` (the #47 merge): no CI, no Image, no CodeQL. The Deploy run of `9b2ca6a` (#49) then saw a newer head and stopped ("main is at 09f6582 now, not 9b2ca6a. Nothing deploys."). So the head of `main` has no CI, no image and no deploy, and `:main` still points at `6c64d29` (#48). Nothing could start those runs again: neither workflow had `workflow_dispatch`. **Changes** - `ci.yml` and `image.yml`: add `workflow_dispatch`. - `image.yml`: push `:<sha>` on every event except `pull_request`, so a manual run on `main` makes the image. - `deploy.yml`: - `promote` accepts an Image run that isn't a `pull_request` (push or manual). The `branches: [main]` filter still limits Deploy to `main`. - "Wait for CI" moves to `.github/scripts/wait-for-ci.sh` and looks for the CI run of the commit with `--branch main`, not `--event push`. A manual CI run counts. A `pull_request` run of the same commit doesn't, because its branch isn't `main`. - `README.md`: "Deploy" says what to run when a push gets no workflows. **After merge:** this merge is a new push, so it should get CI, Image and Deploy normally. If it doesn't, run `gh workflow run ci.yml --ref main` and `gh workflow run image.yml --ref main`. **Checked** - shellcheck (`--shell=sh --enable=check-extra-masked-returns`) and actionlint pass. `oxfmt --check` passes. - The script ran under busybox `ash` with stub `gh`, `timeout` and `sleep`: success after two empty lists, a failed run, a run that never ends (60 polls, then exit 1), a `gh` error, and no arguments (exit 2). - The live query `gh run list --workflow ci.yml --branch main --commit <sha>` finds the push run for `9b2ca6a`, nothing for `09f6582`, and nothing for `3f11cb0` (#49's PR head, which has only a `pull_request` run). **Not changed:** the other inline shell in `deploy.yml` (head check, VM reset, service wait). It can move to scripts in the service audit. I can't confirm why GitHub missed the push. The commit message has no skip marker, and every push workflow is missing, so it looks like a GitHub delivery gap, not this repo's config. <!-- junior-request-attribution:start --> via @dcramer. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0ACFA5JBDX%3A1793830221.324719 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719/?project=4510944073809921) <!-- junior-session-footer:end -->
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #45. It fixes the blockers from the review of #45 (head 411bc6b). Keys do not change: the pin tests pass on b27f425 and on this head.
Changes
knowncap for runs without a token. All sessions of a run count together: at most 10,000 values and 256 Ki characters. A request that would go past the cap fails with 400, and the run ends. A value is never dropped, so no key changes. The count of a session goes away when the session ends or is evicted. Runs with the token have no cap.--max-old-space-size=256 --max-semi-space-size=4.knownand out of templates. V8 cannot compile a pattern with such a literal, so before, every later request of the session failed with 502. On b27f425 those requests never got a key either, so to leave the values out changes no key.extractValues()counts each match when it finds it (anexec()/lastIndexloop), so the cap of 100,000 values fails the request with 400 before memory grows. A test shows that it finds the same values asreplace().normalizeJson()writes the key text without a copy of the body. Thinking blocks are marked in place. A differential check of 20k random cases against b27f425 found 0 key differences.clientGroup()maps IPv4-mapped IPv6 addresses in all forms to IPv4:::ffff:a.b.c.d,::ffff:c000:201, and0:0:0:0:0:ffff:….-target, then the VM reset, then the full apply.Cap data
Before (411bc6b) → after
[a-z]{150}{v:"x"}with 8 MiB ofx"x "× 4 MiThe worst job of a run without a token (16 MiB of
{}) peaks at 714 MB in total, for the main process and the worker together. 10 runs at the cap plus that job: 734 MB.Six new tests fail on 411bc6b and pass here.
RoachServiceConfigdoes not change, sodeploy/gcpneeds no change.via @dcramer.
--
View Session [Sentry]